RGBD Human Activity Recognition by Multi-Modal Context Fusion

We propose a novel complex activity recognition and localization framework which effectively fuses information from both grayscale and depth image channels at multiple levels of the video processing pipeline. In the individual visual feature detection level, depth based filters are applied to the detected human/object rectangles to remove false detections. In the next level of interaction modeling, three dimensional spatial and temporal contexts among human subjects or objects are extracted by integrating information from both grayscale and depth images. Depth information is also utilized to distinguish different types of indoor scenes. Finally, a latent structural model is developed to integrate the information from multiple levels of video processing for activity detection. Extensive experiments on a challenging grayscale + depth human activity database which contains complex interactions between human-human, humanobject and human-surroundings demonstrate the effectiveness of the proposed multi-level grayscale + depth fusion scheme.